Variational Inference for Policy Gradient

Inspired by the seminal work on Stein Variational Inference [2] and Stein Variational Policy Gradient [3], we derived a method to generate samples from the posterior variational parameter distribution by explicitly minimizing the KL divergence to match the target distribution in an amortize fashion. Consequently, we applied this varational inference technique into vanilla policy gradient, TRPO and PPO with Bayesian Neural Network parameterizations for reinforcement learning problems. 1 Parametric Minimization of KL Divergence Suppose we have a random sample from a base distribution ξ ∼ q0(ξ), e.g. q0 = N (0, I), we are able to generate an induced distribution qφ(θ) by the general invertible and differentiable transformation θ = hφ(ξ) (see Appendix A). Our goal is to regard qφ(θ) as a variational distribution to match the true distribution p(θ) such that J = KL(qφ(θ)||p(θ)) is minimized. Lemma 1. H(q) = H(q0) + Eξ∼q0 ( log det ( ∂hφ(ξ) ∂ξ )) (1) with (1), we can have the following identity for KL(qφ(θ)||p(θ)) : KL(q||p) = −H(q)− Eq(θ)(log p(θ)) = −H(q0)− Eξ∼q0 ( log det ( ∂hφ(ξ) ∂ξ )) − Eξ∼q0(log p(hφ(ξ))) = −H(q0)− Eξ∼q0 ( log det ( ∂hφ(ξ) ∂ξ )